docs(review-suite): recheck s2/s3 strata under grader 1.1 for the same surface-in-prose defect - #85
Merged
Merged
Conversation
…e surface-in-prose defect ## Summary - Confirm no v1 raw attempt data survives for s2-solution-simplicity-lens or s3-code-simplicity-lens (searched machine-wide; only unrelated pilot-stratum artifacts remain), matching s1's documented pattern. - Run both strata fresh (40 attempts, 5 runs/case) under the runtime/model pin unchanged (claude 2.1.92, claude-opus-4-6[1m]) and the fixed grader (1.1, commit a247d47). - Re-grade the same fresh raw attempts with the pre-fix grader (85ccf13) as a direct counterfactual: three of the four material cases were already correctly routed to `referred` under the old grader and show no material difference; `setup-service-path-gateway` is materially affected by the same surface-named-only-in-prose defect s1's two cases had, at the referred-vs-false-positive boundary rather than a full blind miss. - Confirm the four clean/negative controls are unchanged (zero false positives, perfect stability, both eras). ## Why - The grader defect fixed in #84 was known to reach s1's two target cases; whether it also reached s2/s3 (whose material cases already showed non-empty referred lists in v1, unlike s1's fully-blind cases) had never been checked. This measurement checks it directly instead of assuming the answer, and finds a real but smaller-scope effect on one case (`setup-service-path-gateway`'s false-positive rate) rather than a clean "unaffected" result. Spend: $2.15 of 40 attempts, against this task's $5.00 ceiling. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prose was invisible to
finding_surfaces()): confirms whether the samedefect reaches
s2-solution-simplicity-lensands3-code-simplicity-lens,neither of which fix(review-suite): grader surface-in-prose defect + v2 s1 re-score comparison #84 touched.
two strata (matching
s1's documented pattern) - a fresh 40-attempt run(5/case, unchanged runtime/model pin:
claude2.1.92,claude-opus-4-6[1m])was required.
pre-fix (
85ccf13) grader shows: three of the four material cases werealready correctly routed to
referredunder the old grader (no materialchange);
setup-service-path-gatewayis materially affected by the samesurface-named-only-in-prose defect, at the referred-vs-false-positive
boundary (2/5 referred + 3/5 false-positive under the old grader vs. 5/5
referred + 0/5 false-positive under the fix) - closely mirroring v1's own
reported
4/5false-positive rate for this exact case.stability, both eras).
Why
s1cases had neverbeen checked for
s2/s3, even though their material cases' non-emptyever_referredlists made them look less likely to be affected. Checkingdirectly (rather than assuming) surfaces a real, smaller-scope effect on one
case's false-positive rate that a report-only before/after comparison would
have missed - this PR's document verifies the claim with a same-attempts,
two-grader-versions counterfactual instead of just diffing two reports.
Scope guardrails
review-suite/evals/v2/; does not touchreview-suite/evals/baseline/v1/,gate-manifest.json,DECISION-RECORD.md,FAILURE-TAXONOMY.md, oraudits/.s1-correctness-orchestrator.decision.
Spend
$2.15 of 40 attempts, against this task's $5.00 hard ceiling.
Test plan
just format- all checks passedjust lint- skills-ref validate + plugin packaging validation passedjust test- 252 review-suite tests + all skill test suites passedreview-code-changecycle (solution-simplicity, correctness,code-simplicity) - all three lenses
clean; correctness independentlyre-derived every load-bearing numeric claim in the new document
(including the counterfactual re-grade) from source data rather than
trusting the PR description.
🤖 Generated with Claude Code